Plenoptic endoscope with fiber bundle

ABSTRACT

A plenoptic endoscope includes a fiber bundle with a distal end configured to receive light from a target imaging region, a sensor end disposed opposite the distal end, and a plurality of fiber optic strands each extending from the distal end to the sensor end. The plenoptic endoscope also includes an image sensor coupled to the sensor end of the fiber bundle, and a plurality of microlenses disposed between the image sensor and the sensor end of the fiber bundle, the plurality of microlens elements forming an array that receives light from one or more of the plurality of fiber optic strands of the fiber bundle and directs the light onto the image sensor. The plurality of microlens elements and the image sensor together form a plenoptic camera configured to capture information about a light field emanating from the target imaging region.

PRIORITY CLAIM

This application claims priority to U.S. Provisional Application62/648,060, filed on Mar. 26, 2018, the entire contents of which areincorporated herein by reference.

BACKGROUND

Plenoptic cameras (sometimes referred to more generally as light fieldcameras), capture information on a light field emanating from an objector surface with a lens and image sensor configuration that capturesinformation on both the intensity and the direction of light. Aplenoptic endoscope facilitates light field imaging of an object orsurface inside the human body.

SUMMARY

In one aspect, the present disclosure features a plenoptic endoscopeincluding a fiber bundle with a distal end configured to receive lightfrom a target imaging region, a sensor end disposed opposite the distalend, and a plurality of fiber optic strands each extending from thedistal end to the sensor end. The plenoptic endoscope also includes animage sensor coupled to the sensor end of the fiber bundle, and aplurality of microlenses disposed between the image sensor and thesensor end of the fiber bundle. The plurality of microlens elements forman array that receives light from one or more of the plurality of fiberoptic strands of the fiber bundle and directs the light onto the imagesensor. The plurality of microlens elements and the image sensortogether form a plenoptic camera configured to capture information abouta light field emanating from the target imaging region.

In some instances, the plenoptic endoscope includes a main lens assemblyarranged at the distal end of the fiber bundle, the main lens arrangedto direct light from the target imaging region through the plurality offiber optic strands.

In some instances, the main lens assembly includes an objective lensdefining a field of view of the light directed through the plurality offiber optic strands.

In some instances, the fiber bundle is flexible. In some instances, thefiber bundle is configured to be inserted into an animal or human body.

In some instances, the fiber bundle includes a proximal portion and adistal portion coupled at a joint along a length of the fiber bundle,the joint enabling angular movement of the distal portion with respectto the proximal portion about the joint. In some instances, the jointincludes an electric motor configured to cause the angular movement ofthe distal portion with respect to the proximal portion about the joint.

In some instances, the fiber bundle includes a plurality of separate,individually flexible arms, each of the plurality of arms includes asubset of the plurality of fiber optic strands. In some instances, theimage sensor includes a plurality of image sensing regions, and eachimage sensing region is arranged to receive light from one of theplurality of separate individual flexible arms. In some instances, afirst image sensing region of the plurality of image sensing regions isconfigured to sense at least a first wavelength of light, and a secondimage sensing region is configured to sense at least a second wavelengthof light that is not sensed by the first image sensing region. In someinstances, the first image sensing region is configured to sense atleast visible light, and the second image sensing region is configuredto sense at least infrared light.

In some instances, the plenoptic endoscope includes at least one lightsource and a subset of the plurality of fiber optic strands of eachflexible arm forms a light guide arranged to direct light from the lightsource to the distal end of each flexible arm to illuminate at least aportion of the target imaging region. The distal end of a first flexiblearm is configured to direct light of at least a first wavelength oflight and the distal end of a second flexible arm is configured todirect light of at least a second wavelength of light that is notdirected by the first flexible arm. In some instances, the image sensorincludes a plurality of image sensors, where each image sensor isarranged to receive light from one of the plurality of separateindividual flexible arms.

In some instances, the plenoptic includes a light source, and a subsetof the plurality of fiber optic strands forms a light guide arranged todirect light from the light source to the distal end of the fiber bundleto illuminate the target imaging region.

In some instances, each of the plurality of microlens elements isarranged to receive light from one or more of the plurality of fiberoptic strands of the fiber bundle and to direct the light onto the imagesensor.

In some instances, one or more of the plurality of microlens elements isarranged to receive light from one of the plurality of fiber opticstrands of the fiber bundle and direct the light onto the image sensor.

In some instances, the plenoptic endoscope includes one or moreprocessing devices operatively coupled with the image sensor andconfigured to calculate surface image data for a surface in the targetimaging region. In some instances, the one or more processing devices isconfigured to generate representations of a depth map based on thesurface image data and a photographic image.

In another aspect, this document features a method of presenting animage on a display of a plenoptic endoscope. The method includesreceiving, at an image sensor, light from a target imaging region, wherethe light is captured using a plurality of microlens elements disposedat a sensor end of a fiber bundle including a plurality of fiber opticstrands, and the plurality of microlens elements and the image sensortogether form a plenoptic camera to capture both intensity and directioninformation of the light. The method includes computing, based on anoutput of the image sensor, a representation of a portion of the targetimaging region, computing, based on the output of the image sensor, adepth map associated with the portion of the target imaging region, andpresenting the image on the display of the plenoptic endoscope, theimage being generated from the representation of the portion and theassociated depth map associated with the surface. In some instances, theimage is a multi-focus image.

DESCRIPTION OF DRAWINGS

FIGS. 1A and 1B are schematics of two different types of plenopticimaging systems configured to be used in a plenoptic endoscope.

FIG. 2 is a schematic diagram of an example of a plenoptic endoscopewith a fiber bundle.

FIG. 3 is a schematic diagram of the internal structure of a fiberbundle of the plenoptic endoscope of FIG. 2.

FIG. 4 is a block diagram of an example of a plenoptic endoscope andsubsequent image processing steps performed on images captured using theplenoptic endoscope.

FIG. 5 is a schematic diagram of an example of a plenoptic endoscopewith a flexible fiber bundle.

FIG. 6 is a schematic diagram of an example of a plenoptic endoscopewith a fiber bundle having an articulated joint in the fiber bundle.

FIG. 7 is a schematic diagram of an example of a plenoptic endoscopewith a bifurcated fiber bundle.

FIG. 8 is a block diagram of an example of a steerable plenopticendoscope system.

FIG. 9 is a block diagram of a remotely operated steerable plenopticendoscope system.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

This document describes a plenoptic endoscope that combines an endoscopehaving a fiber bundle configured to be inserted into a body and relayingimages of areas therein with a plenoptic camera system. The plenopticcamera system captures direction and intensity information about thelight received via the fiber bundle to create a three-dimensional depthmap of the imaged structures and surfaces viewed by the endoscope.Aspects of the plenoptic endoscopes disclosed herein enable generationof quantitative information regarding the size and position of internalstructures, which in turn may aid the operation of surgical instruments(e.g., that in a telesurgical system), operating in the view of theplenoptic endoscope. For example, a plenoptic endoscope imaging a targetstructure and a tool can calculate the spatial relationship (e.g.,distance and orientation) between the tool and the target structure inorder to guide the tool's interaction with the target structure.

Plenoptic cameras (sometimes referred to more generally as light fieldcameras), capture light emanating from an object or surface with a lensand image sensor configuration that captures both the intensity anddirection of the light. The light field associated with the capturedlight can then be resolved from the information captured by the imagesensor. A plenoptic endoscope utilizes such light field photographytechniques, for example, to generate a three-dimensional (3D) and/orspectral image of an object or surface (e.g., inside the human body).Plenoptic cameras can be of different types. A standard plenoptic cameraincludes an array of multiple microlenses placed one focal length awayfrom the image plane of a sensor. A focused plenoptic camera includes amicrolens array in front of, or behind the focal plane of the main lens,and the light field is sampled in a way that trades angular resolutionfor higher spatial resolution. In a focused plenoptic camera, images canbe post focused with a much higher spatial resolution than with imagesfrom the standard plenoptic camera.

A plenoptic endoscope is an endoscopic imaging system that includes aplenoptic camera. The imaging system may include relay optics at theterminal end of an endoscope arranged to form an image of the surface orobject being investigated. In some implementations, the plenoptic cameraincludes an image sensor (e.g., a CCD or CMOS digital camera sensor),and a microlens array positioned at a location between the image sensorand the imaging system. Based on the known optical characteristics ofthe microlens array and the positioning with respect to the imagesensor, light received by the image sensor from the relay optics isdeconstructed in a light field that includes depth information from eachimaged surface with respect to the relay optics. In this manner, a 3Dimage or depth map of the surface of object can be created from thecaptured light field information.

In robotic surgery, visualization and perception of the surgical site isan important aspect, and quantitative imaging (e.g., a depth map or 3Dimage) can improve surgical outcome by providing meaningful additionalinformation to the surgeon. Several technologies are currently availablefor quantitative 3D imaging in the operating room, such as magneticresonance imaging (MRI), computed tomography (CT) scans, and 3Dultrasound. However, these are often difficult to integrate intoreal-time surgery. One solution is to use stereo endoscopic images togenerate 3D models, and to reconstruct and track surfaces of soft tissueusing such 3D models. Since surgeons already use stereoscopic endoscopesfor depth perception during minimally invasive surgery (MIS), thehardware is already available. However, such quantitative imagingentails calibration of endoscopes and stereo matching, which may bechallenging in some cases. In some cases, plenoptic endoscopes can beused to generate a 3D visualization by leveraging the light fieldphotography techniques used in such endoscopes. In addition to creatinga 3D depth map of the field, plenoptic systems can provide a focusedimage at different depths; this is not something a stereo camera coulddo.

A representative example of a plenoptic endoscope includes a plenopticimaging system integrated with an endoscope constructed from a bundle ofoptical fiber strands. The bundle of optical fiber strands (hereinafterreferred to simply as a ‘fiber bundle’) is used both as a light guide toilluminate a target imaging region, and to transmit the light emanatingfrom the target imaging area, from a distal end (e.g., the endpositioned at the imaging region inside the body) to a sensor end (e.g.,the end transmitting the light from the fiber bundle to the imagingsystem) of the endoscope. In some instances, a group of the fiber opticstrands are used for illumination, while a different group of the fiberoptic strands are used for relaying the light from the target imagingregion end to the plenoptic camera. The plenoptic endoscope includes anarray of microlenses between the sensor end of the fiber bundle and animage sensor of the imaging system. In some examples, a second array ofmicrolenses is arranged on the distal end of the fiber bundle to focusthe incoming light into the fiber bundle. In some implementations, aseries of microlenses of appropriate powers may be used in place of asingle microlens. In some examples, a relay lens assembly is alsoprovided at the distal end of the fiber bundle to define the field ofview of the light received by the second array of microlenses. With thisplenoptic endoscope arrangement, a calibration algorithm can bedeveloped, based on, for example, the relative positioning of the imagesensor and the microlenses, and their optical characteristics, toconvert the data received by the image sensor into a 2D image and/or a3D depth map of the view of the plenoptic endoscope.

Certain aspects of the plenoptic endoscope embodiments provide a numberof advantages over existing systems. In some implementations, byproviding flexible fiber optic strands in the fiber bundle, that thetechnology described herein enables construction of a flexible fiberbundle for use in an endoscope. The flexible fiber bundle can includeboth (i) light field fibers carrying a light from an imaging region to aplenoptic camera and, (ii) light guide fibers carrying illumination intothe imaging region packaged together as a single flexible unit. In someimplementations, the flexible plenoptic endoscope described herein maybe used for 3D depth sensing and surface reconstruction in endoluminalsurgery/interventions at portions of the human body that are challengingto reach with rigid endoscopes.

In some implementations, a plenoptic endoscope includes a wristed orarticulated fiber bundle with 3D depth sensing and surfacereconstruction capabilities. In this example, the fiber bundle iscoupled to a mechanized wristed endoscope tip with 3D depth sensing andsurface reconstruction capabilities. Flexibility of the fiber bundleallows the fiber bundle to bend with the articulation of the mechanizedwrist, thereby potentially facilitating an enhanced field of view ascompared to a non-wristed endoscope.

In some implementations, a plenoptic endoscope includes a bifurcatedfiber bundle with 3D depth sensing and surface reconstructioncapabilities. In such implementations, the fiber bundle is bifurcatedinto two or more separate subsets of fiber bundles, each potentiallyimaging/mapping a different portion of the surgical site. In someinstances, the bifurcated plenoptic endoscope may enable an extendedfield of view and/or enable a user to look sideways or backwardsdepending on the dexterity of each distal section of the bifurcatedfiber bundle.

FIGS. 1A and 1B are schematics of two different types of plenopticimaging systems. FIG. 1A shows a plenoptic camera that includes anobjective lens 30, an image sensor 11, and a microlens array 20 arrangedbetween the image sensor 11 and the objective lens 30. The microlensarray 20 includes individual microlenses 21 forming a 2D matrix. FIG. 1Arepresents a standard plenoptic system, where the objective lens 30 hasa fixed focal plane 39 on which the microlens array 20 is arranged. Inoperation, the objective lens 30 directs light along a cone 31 to thefocal plane 39 and each microlens 21 projects a sampling 32 of acorresponding cone 31 onto a portion of the image sensor 11. Themultiple microlenses together allow for capturing images of the sameobject as if viewed from different directions. As a result of thedifferent spatial relationship between each microlens 21 and theobjective lens 30, the image data captured by the image sensor 11 isable to sense multiple spatially-shifted views of the target imagingregion, and thereby capture information regarding the depth of thesource of the light rays directed from the objective lens 30 to theimage sensor 11. FIG. 1B shows an alternative plenoptic camera 10configuration, whereby the microlens array 20 is positioned between thefixed focal plane 39 of the objective lens 30 and the image sensor 11,such that the portion of light 33 captured by each microlens 21 isdefocused, and each microlens 21 projects a focused light field 34 ontothe image sensor 11. More details about plenoptic cameras may be foundin the reference “Light Field Photography with a Hand-held PlenopticCamera,” Stanford Tech Report CTSR 2005-02, the entire content of whichis incorporated herein by reference.

FIG. 2 is a schematic diagram of an example of a plenoptic endoscope 100with a fiber bundle 200. The plenoptic endoscope 100 includes aplenoptic camera system 10 operatively coupled to the fiber bundle 200at a sensor end 220 of the fiber bundle 200. An objective lens 30 isdisposed at an opposite, distal end 230 of the fiber bundle 200 tocapture light from a target imaging region into the fiber bundle 200. Inoperation, the fiber bundle transmits the light received through theobjective lens 30 to the image sensor 11 of the plenoptic camera system10. In some instances, the plenoptic camera system 10 is either astandard plenoptic arrangement 104 or a focused plenoptic arrangement105. The fiber bundle 200 is constructed from a plurality of fiber opticstrands 250, which are grouped to create a pathway for light to travelto between the distal end 230 and the sensor end 220. A microlens array20 is arranged at the sensor end 220 of the fiber bundle 200. Thismicrolens array 20 is positioned with respect to the image sensor 11 ofthe plenoptic camera system 10 to enable the image sensor to resolve thelight field associated with the light captured by the objective lens 30and transmitted through the fiber bundle 200. In some instances, thefiber bundle 200 includes a second microlens array 320 (shown in FIG. 3)between the objective lens 30 and the fiber optic strands at the distalend 230. In some instances, a portion of the fiber bundle 200 isflexible, and another portion of the fiber bundle is substantiallyrigid.

In operation, the plenoptic endoscope 100 is used to image anatomyinside a human or animal body, by inserting the distal end 230 of thefiber bundle 200 into the body and relaying light captured by theobjective lens 30 at the distal end 230 to the plenoptic camera system10 at the sensor end 220. In this configuration, the plenoptic endoscope100 enables generation of both a photographic image based on the view ofthe objective lens 30, which can be re-focused at different depths, aswell as a depth map of the surfaces and structures visible in the fieldof view. In some implementations, the plenoptic endoscope 100 may beused outside the body as well. In some instances, and especially foruses inside a body, the fiber bundle 200 also includes one or morefibers acting as a light guide to direct light from a source outside thebody to the distal end 230 to provide illumination to the anatomy orsurfaces being imaged. In some example, the fiber bundle 200 is sealedinside an outer casing (not illustrated) to enable safe insertion of thedistal end 230 of the fiber bundle 200 into an animal or human body.

FIG. 3 is a schematic diagram of the internal structure of a fiberbundle of the plenoptic endoscope 100 of FIG. 2. In FIG. 3, the fiberbundle 200 includes an objective lens 30 at the distal end 230, whichdefines a field of view 38 from which light is directed by the objectivelens 30 to the image sensor 11 of the plenoptic camera system 10 via thefiber bundle 200. As detailed above with respect to FIG. 2, the fiberbundle 200 includes a plurality of fiber optic strands 250 spanning thelength of the fiber bundle 200. However, multiple configurations arepossible, and three exemplary arrangements are depicted in FIG. 3. Thefiber bundle 200 includes a microlens array 320 at the distal end 230and a microlens array 20 at the sensor end 220. In some implementations,the fiber optic strands 250 span the length of the fiber bundle 200between the two microlens arrays 20, 320. In a first configuration, eachindividual fiber optic strand 351 is arranged with respect to themicrolens arrays 20, 320 such that a single microlens 321 a at thedistal end 230 is optically coupled to a single fiber optic strand 351and, similarly, a single microlens 321 b at the sensor end 220 isoptically coupled to the same single fiber optic strand 351. In thisfirst configuration, light received by each microlens 321 a at thedistal end 230 is directed (as illustrated by arrow 350) to acorresponding microlens 321 b at the sensor end 220 by the single fiberoptic strand 351.

In a second configuration, a fiber optic strand 353 is arranged withrespect to the microlens arrays 20, 320 such that a group of microlensesat the distal end 230 are optically coupled to a fiber optic strand 353and, similarly, a group of microlenses at the sensor end 220 areoptically coupled to the same fiber optic strand 353. In this secondconfiguration, light received by a group of microlenses at the distalend 230 is directed (as illustrated by arrow 350) to a correspondinggroup of microlenses at the sensor end 220.

In a third configuration, a group of fiber optic strands 352 a-352 d arearranged with respect to the microlens arrays 20 such that a singlemicrolens 321 c at the distal end 230 is optically coupled to the groupof fiber optic strands 352 a-352 d and, similarly, a single microlens321 d at the sensor end 220 is optically coupled to the same group offiber optic strands 352 a-352 d. In this third configuration, lightreceived by each microlens 321 c at the distal end 230 is directed (asillustrated by arrow 350) to a corresponding microlens 321 d at thesensor end 220 by the group of fiber optic strands 352 a-d. A fiberbundle 200 may be constructed from any combination of the differentfiber optic strand configurations.

Continuing to refer to FIG. 3, the fiber bundle 200 includes one or morefiber optic strands configured as illumination fibers 360 (e.g., lightguides) to transmit light from a source of illumination (notillustrated) to the region within the field of view 38 of the objectivelens 30. In some instances, the illumination fibers 360 are positionedbetween the light sensing fibers (e.g., fiber optic strands 351, 352a-352 b, 353) carrying sensed light to the plenoptic camera system 10.In some instances, the light guides exit the fiber bundle 200 at thedistal end 230 in small gaps between each microlens 21 of the microlensarray 20. A distribution of the illumination fibers 360 among theillumination fibers can be adjusted to minimize effects of specularlight in the field of view 38 of the endoscope. In some instances, theillumination fibers 360 achieve an even distribution of light at thesurgical site, for example by distributing them circumferentially aroundthe light sensing fibers. The illumination fibers 360 can be dynamicallycontrolled to adjust the lighting in the field of view 38 to createimages with more uniform illumination. As an example, if the right sideof the image is closer to the objective lens 30, less illumination mightbe required for that area, compared to the left side of the image, whichhas a greater distance from the objective lens 30. In other instances,the illumination fibers 360 are positioned outside of the light sensingfibers.

FIG. 4 is a block diagram of an example of a plenoptic endoscope system40 and subsequent image processing steps. The plenoptic endoscope system40 includes a fiber bundle 200 attached to a plenoptic camera system 10and an image processor 401. In some instances, the objective lens 30 ofthe fiber bundle 200 includes a set of relay lenses 35 a-35 d (35, ingeneral) positioned between beyond the microlens array 20 at the distalend 230 of the fiber bundle 200 to extend the distance between theobjective lens 30 and the microlens array 320. In some instances, theobjective lens 30 is positioned at the distal end 230 without a set ofrelay lenses 35, and in other instances, the microlens array 320 at thedistal end 230 is configured to operate as an objective lens. In someinstances, the relay lenses 35 enable other optical changes of the lighttransmitted to the fiber bundle 200, such as changing the focal lengthby changing the field of view 38. A number of optical configurations canbe used to deliver light from an object 48 positioned within a field ofview 38 to the fiber optic strands 250, which in turn deliver the lightto the microlens array 20 at the sensor end 220. From the microlensesarray 20 at the sensor end 220, a defocused or focused image isprojected from each microlens 21 onto the image sensor 11 of theplenoptic camera system 10. The image sensor 11, in some instances, isan array of photosensitive pixels 12. The image sensor 11 generatesimage data 400 that is passed onto to a processor 401. The input to theprocessor 401 are images obtained from the light field in relation toeach microlens 21. The first step, reconstruction 410, is to match localintensity in nearby image regions (microlens array sub-images). Oncecommon image features, for example, corners, which correspond to thesame physical object, are identified, the pixels are back projected ontothe light field (multi-image triangulation) to reconstruct a local 3Dpoint cloud 420. The local 3D point cloud 420 from nearby microlensarray 320 sub-images overlap, from which the common areas are removed.The result is a reconstructed 3D surface 430 of the object 48 that canbe projected on to a total focus image 440 to create a regular image450. In some implementations, the foregoing process may be calibratedtin accordance with a cost function associated with the above-referencedback-projection. For example, the cost function can be a function ofdistances between lenses in the microlens array 320, the distance of themicrolens array 320 to the distal end 230 of the fiber bundle 200,and/or the distance of the microlens array 320 to image sensor 11. Inaddition, for a metric point cloud, the intrinsic parameters of eachmicrolens 21, for example, focal length, is calculated.

FIG. 5 is a schematic diagram of an example of a plenoptic endoscope 50with a flexible fiber bundle 500. The plenoptic endoscope 50 includes aplenoptic camera system 10 with the flexible fiber bundle 500 arrangedto direct light captured by an objective lens 30 at a distal end 230 ofthe flexible fiber bundle 500 to an image sensor 11 of the plenopticcamera system 10. The flexible fiber bundle 500 includes a firstmicrolens array 20 at a sensor end 220 and, in some instances, a secondmicrolens array (as shown in FIG. 3) at the distal end 230. Becauseindividual fiber optic strands 250 are able to flex and bend, theflexible fiber bundle 500 is constructed to allow the flexibility of thefiber optic strands 250 to permit flexion of the entire flexible fiberbundle 500 as illustrated. In operation, the flexible fiber bundle 500enables the plenoptic camera system 10 to remain stationary while theobjective lens 30 is oriented in an arbitrary position. In eacharbitrary position of the flexible fiber bundle 500, the distancebetween the first and second microlens arrays is approximately equal dueto the reluctance of the individual fiber optic strands 250 to compressor stretch significantly. As a result, a precise distance between eachmicrolens 21, as provided by the calibration process, can be maintained.FIG. 5 is only an example illustration of a bend in a flexible fiberbundle 500, and any apparent change in the length of the fiber opticstrands 250 shown is not intended to reflect any real materialproperties.

FIG. 6 is a schematic diagram of an example of a plenoptic endoscope 60with an articulated fiber bundle 600 having a joint 601 disposed alongthe length of the articulated fiber bundle 600, the joint coupling aproximal portion 652 of the fiber bundle to a distal portion 653 of thefiber bundle. The plenoptic endoscope 60 includes a plenoptic camerasystem 10 with the articulated fiber bundle 600 arranged to direct lightfrom an objective lens 30 at a distal end 230 of the articulated fiberbundle 600 to an image sensor 11 of the plenoptic camera system 10. Thearticulated fiber bundle 600 includes a first microlens array 20 at asensor end 220 and, in some instances, a second microlens array (asshown in FIG. 3) at the distal end 230. Because individual fiber opticstrands 250 are able to flex and bend, the articulated fiber bundle 600is constructed to allow the flexibility of the fiber optic strands 250to permit a joint 601 to bend the articulated fiber optic bundle 600 asillustrated. In operation, the articulated fiber bundle 600 enables theplenoptic camera system 10 to remain stationary while the distal portion653 of the fiber bundle is oriented at an angle within a range of anglesas afforded by the movement of the distal portion 653 of the fiberbundle 600 with respect to the proximal portion 652 of the fiber bundle600 around the joint 601. In some instances, the joint 601 includes amotor 660 controlling the position of the distal portion 653 of thefiber bundle 600 with respect to the proximal portion 652 of the fiberbundle 600. In some instances, the fiber optic strands 250 bend (asindicated at point 650) as they pass through the joint 601. In someinstances, the joint 601 is articulated via a cable mechanism around thefiber bundle 600. In some instances, the joint 601 is articulated via acable mechanism (not shown) similar to other wristed instruments, forexample a vessel sealer. In some instances, the fiber optic strands arespooled (as indicated by loop 651) or are otherwise bunched near thejoint 601 such that the joint's 601 articulation of the articulatedfiber bundle 600 maintains an approximately equal distance between thefirst and second microlens arrays along each of the individual fiberoptic strands 250. Any spooling mechanism can be configured inaccordance with any curvature limitations associated with the fibers.

FIG. 7 is a schematic diagram of an example of a plenoptic endoscope 70with a bifurcated fiber bundle 700. The plenoptic endoscope 70 includesa plenoptic camera system 10 with the bifurcated fiber bundle 700arranged to direct light from two objective lenses 30 at separate distalends 730 a, 730 b of the bifurcated fiber bundle 700 to an image sensor11 of the plenoptic camera system 10. The bifurcated fiber bundle 700disposed at the distal end of the endoscope includes a single microlensarray 20 at a sensor end 220 and, in some instances, a second microlensarray (as shown in FIG. 3) at each of the distal ends 730 a, 730 b. Insome instances, the bifurcation refers to one subset of fiber strandsforming a first arm, and a different subset of fiber strands forming asecond arm. In some implementations, the bifurcated fiber bundle 700 caninclude a set of fiber optic strands 250 bifurcated into two separatefiber bundles 701 a, 702 b, with each having a distal end 730 a (or 730b) with an objective lens 30. In some implementations, the light fromdifferent fiber bundles may be sensed using different combinations ofsensors and/or filter. For example, appropriate optical filters can beused with one fiber bundle 701 a for sensing visible light, whereasemission filters can be used with another fiber bundle 701 b for sensinginfrared light. This in turn can enable, for example, overlayinginformation about a tumor (e.g., as captured in the infrared range usingthe fiber bundle 701 b) onto a visible light image (e.g., as captured inthe visible range using the fiber bundle 701 b). In some instances, thetwo separate fiber bundles 701 a, 702 b are coupled to separate imagesensors (not shown), where the separate image sensors are configured tobe responsive to at least partially non-overlapping frequency ranges.

In operation, the bifurcated fiber bundle 700 enables the plenopticcamera system 10 to remain stationary while each of the objective lens30 are oriented in different positions. In some instances, the twoobjective lenses 30 may be oriented to view the same object 71, and, asshown, but from different viewpoints, for example to overlay anatomyreconstructed from one view onto anatomy reconstructed on the otherview. For example, each objective lens may be oriented to observe adifferent surface 78 a, 78 b of the object 71 in their respective fieldsof view 38. In some cases, the respective fields of view of the twoobjective lenses may be at least partially overlapping. For example, aportion of the surface 78 c may be within the field of view 38 of bothobjective lenses of the bifurcated fiber bundle 700. Therefore, in someimplementations, the bifurcated fiber bundle 700 may enable (1) a largerfield of view for viewing an object 71, and/or (2) calculation of acorrelation between overlapping portions of the two fields of view 38.In some instances, the correlation is used to create a depth map of thesurface 78 a, 78 b, 78 c of the object 71. In some instances, becausethe overlapped surface 78 c is viewed by two objective lensed, a highspatial resolution depth mapping of the surface 78 c is possible.Further, the bifurcated endoscope may be used to enable differentviewing directions (e.g., sideways, backwards etc.) depending, forexample, on the dexterity of each distal arm of the bifurcated fiberbundle.

While the description herein specifically mentions a bifurcated fiberbundle with two branches or arms, more branches or arms are also withinthe scope of this disclosure. For example, three or more flexible arms,each including a separate subset of the fiber bundle, can be implementedsuing the technology described herein. In some implementations, one ofthe subsets of the fiber bundle may be used for transmitting light froma light source, and one or more of other subsets of the fiber bundle maybe used to transmit captured light back to the image sensor.

FIG. 8 illustrates, as an example, a plenoptic endoscope 80 including asteerable fiber bundle 110, a plenoptic camera system 10 with an imagesensor 11, an image processor 140 coupled to the plenoptic camera system10, a plurality of fiber optic strands 250 directing light from anobjective lens 30 at the distal end 230 of the steerable fiber bundle110 to the image sensor 11, a display processor 150, a primary displayscreen 151, an auxiliary display screen 152, a main processor 160, andmemory 161. Although shown as separate units, the image processor 140,display processor 150, and main processor 160 may be implemented in asingle processor or their respective functions distributed among aplurality of processors, where each of such processors may beimplemented as hardware, firmware, software or a combination thereof. Asused herein, the term processor is understood to include interface logicand/or circuitry for translating and/or communicating signals intoand/or out of the processor as well as conventional digital processinglogic. The memory 161 may be any memory device or data storage system asconventionally used in computer systems. The primary and auxiliarydisplay screens, 151 and 152, are preferably computer monitors capableof displaying three-dimensional images to an operator of the plenopticendoscope 80. However, for cost or other considerations, either or bothof the primary and auxiliary display screens, 151 and 152, may be astandard computer monitor capable of only displaying two-dimensionalimages.

The steerable fiber bundle 110 has a flexible body 114, a steerable tip112 at its distal end 230, and a hand-operable handle 116 at itsproximal end 115. Control cables (not shown) or other control meanstypically extend from the handle 116 to the steerable tip 112 so thatthe tip 112 may be controllably bent or turned as shown for example bydotted line versions of the bent tip 112.

The image sensor 11 of the plenoptic camera system 10 outputs image datato be processed by the image processor 140 and/or display processor 150and displayed on the primary display screen 151, auxiliary displayscreen 152, and/or other display means according to the various aspectsof the invention as described herein. The plenoptic camera system 10 mayalso be single or multi-spectral that captures image data in the visibleor infrared/ultraviolet spectrum. Thus, any image sensor 11 referred toherein may be any one or a combination of these and other imagingtechnologies. In some implementations, a portion of an image sensor canbe configured to sense light in a first portion of the spectrum (e.g.,visible light range) while another portion of the sensor is configuredto sense light in a different, second portion (e.g., infra-red) that isat least partially non-overlapping with the first portion. One of aplurality of fiber optic strands 250 may be coupled at its sensor end220 to a light source (not shown) for illumination purposes at thedistal end 230. Another of the plurality of fiber optic strands 250 maybe configured with position and bend or shape sensors such as FiberBragg Gratings (or other strain sensors such as those employing Rayleighscattering) distributed along the length of the steerable fiber bundle110 so that light passing through these fiber optic strands 250 areprocessed by a sensor processor (not illustrated) to determine a currentpose and shape of the steerable fiber bundle 110.

FIG. 9 illustrates, as an example, an alternative embodiment of theplenoptic endoscope 90 in which the handle 116 is replaced by anelectromechanical interface 170, controller 180, and input device 190for teleoperating the steerable fiber bundle 110 of the plenopticendoscope 90. The interface 170 includes actuators for actuating cablesin the steerable fiber bundle 110 to steer its tip 112 as well as anactuator for moving the entire steerable fiber bundle 110 forward andbackward so that it may be inserted into and retracted out of a patientthrough an entry port such as a natural body orifice or a surgeoncreated minimally invasive incision. In addition, the interface 170 mayinclude an actuator for rotating the steerable fiber bundle 110 aboutits central longitudinal axis. The controller 180 is preferablyimplemented as hardware, firmware or software (or a combination thereof)in the same one or more computer processors as the processors 140, 150,and 160, or a different computer processor. The flexible body 114 may bepassively or actively bendable. The plenoptic endoscope 90 can also be ahybrid of the above two examples.

The functionality described herein, or portions thereof, and its variousmodifications (hereinafter “the functions”) can be implemented, at leastin part, via a computer program product, e.g., a computer programtangibly embodied in an information carrier, such as one or morenon-transitory machine-readable media or storage device, for executionby, or to control the operation of, one or more data processingapparatus, e.g., a programmable processor, a DSP, a microcontroller, acomputer, multiple computers, and/or programmable logic components.

A computer program can be written in any form of programming language,including compiled or interpreted languages, and it can be deployed inany form, including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment. Acomputer program can be deployed to be executed one or more processingdevices at one site or distributed across multiple actions associatedwith implementing all or part of the functions can be performed by oneor more programmable processors or processing devices executing one ormore computer programs to perform the functions of the processesdescribed herein. All or part of the functions can be implemented as,special purpose logic circuitry, e.g., an FPGA, an ASIC(application-specific integrated circuit), GPU (graphics processingunit), a Tensor Processing Unit (TPU), and/or another parallelprocessing unit.

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. Components of a computer include aprocessor for executing instructions and one or more memory devices forstoring instructions and data.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinvention or of what may be claimed, but rather as descriptions offeatures that may be specific to particular embodiments of particularinventions. Certain features that are described in this specification inthe context of separate embodiments can also be implemented incombination in a single embodiment. Conversely, various features thatare described in the context of a single embodiment can also beimplemented in multiple embodiments separately or in any suitablesubcombination. Moreover, although features may be described herein asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a sub combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous, for example, in image reconstruction. Moreover, theseparation of various system modules and components in the embodimentsdescribed herein should not be understood as requiring such separationin all embodiments, and it should be understood that the describedprogram components and systems can generally be integrated together in asingle product or packaged into multiple products.

Elements described in detail with reference to one embodiment,implementation, or application optionally may be included, wheneverpractical, in other embodiments, implementations, or applications inwhich they are not specifically shown or described. For example, if anelement is described in detail with reference to one embodiment and isnot described with reference to a second embodiment, the element maynevertheless be claimed as included in the second embodiment. Thus, toavoid unnecessary repetition in the following description, one or moreelements shown and described in association with one embodiment,implementation, or application may be incorporated into otherembodiments, implementations, or aspects unless specifically describedotherwise, unless the one or more elements would make an embodiment orimplementation non-functional, or unless two or more of the elementsprovide conflicting functions.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

What is claimed is:
 1. A plenoptic endoscope, comprising: a fiber bundlecomprising: a distal end configured to receive light from a targetimaging region, a sensor end disposed opposite the distal end, and aplurality of fiber optic strands each extending from the distal end tothe sensor end; an image sensor coupled to the sensor end of the fiberbundle; and a plurality of microlenses disposed between the image sensorand the sensor end of the fiber bundle, the plurality of microlensesforming an array that receives light from one or more of the pluralityof fiber optic strands of the fiber bundle and directs the light ontothe image sensor, wherein the plurality of microlenses and the imagesensor together form a plenoptic camera configured to captureinformation about a light field emanating from the target imagingregion.
 2. The plenoptic endoscope of claim 1, comprising a main lensassembly arranged at the distal end of the fiber bundle, the main lensassembly arranged to direct light from the target imaging region throughthe plurality of fiber optic strands.
 3. The plenoptic endoscope ofclaim 2, wherein the main lens assembly comprises an objective lensdefining a field of view of the light directed through the plurality offiber optic strands.
 4. The plenoptic endoscope of claim 1, wherein thefiber bundle is flexible.
 5. The plenoptic endoscope of claim 1, whereinthe fiber bundle comprises a proximal portion and a distal portioncoupled at a joint along a length of the fiber bundle, the jointenabling angular movement of the distal portion with respect to theproximal portion about the joint.
 6. The plenoptic endoscope of claim 5,wherein the joint comprises an electric motor configured to cause theangular movement of the distal portion with respect to the proximalportion about the joint.
 7. The plenoptic endoscope of claim 1, whereinthe fiber bundle comprises a plurality of separate, individuallyflexible arms, each of the plurality of arms comprising a subset of theplurality of fiber optic strands.
 8. The plenoptic endoscope of claim 7,wherein the image sensor comprises a plurality of image sensing regions,wherein each image sensing region is arranged to receive light from oneof the plurality of separate, individually flexible arms.
 9. Theplenoptic endoscope of claim 8, wherein a first image sensing region ofthe plurality of image sensing regions is configured to sense at least afirst range of wavelengths of light, and wherein a second image sensingregion is configured to sense at least a second range of wavelength oflight, the first range and the second range being at least partiallynon-overlapping.
 10. The plenoptic endoscope of claim 9, wherein thefirst image sensing region is configured to sense visible light, andwherein the second image sensing region is configured to sense infraredlight.
 11. The plenoptic endoscope of claim 7, comprising at least onelight source, wherein a subset of the plurality of fiber optic strandsof each flexible arm forms a light guide arranged to direct light fromthe light source to the distal end of each flexible arm to illuminate atleast a portion of the target imaging region, wherein a first opticalfilter disposed at the distal end of a first flexible arm is configuredto pass light in a first wavelength range, range and wherein a secondoptical filter disposed at the distal end of a second flexible arm isconfigured to pass light in a second wavelength range that is at leastpartially non-overlapping with the first wavelength range.
 12. Theplenoptic endoscope of claim 7, wherein the image sensor comprises aplurality of pixel arrays, wherein each pixel array is arranged toreceive light from one of the plurality of separate, individuallyflexible arms.
 13. The plenoptic endoscope of claim 1, wherein the fiberbundle is configured to be inserted into an animal or human body. 14.The plenoptic endoscope of claim 1, comprising a light source, andwherein a subset of the plurality of fiber optic strands forms a lightguide arranged to direct light from the light source to the distal endof the fiber bundle to illuminate the target imaging region.
 15. Theplenoptic endoscope of claim 1, wherein each of the plurality ofmicrolenses is arranged to receive light from one or more of theplurality of fiber optic strands of the fiber bundle and to direct thelight onto the image sensor.
 16. The plenoptic endoscope of claim 1,wherein one or more of the plurality of microlenses is arranged toreceive light from one of the plurality of fiber optic strands of thefiber bundle and direct the light onto the image sensor.
 17. Theplenoptic endoscope of claim 1, further comprising one or moreprocessing devices operatively coupled with the image sensor andconfigured to calculate surface image data for a surface in the targetimaging region.
 18. The plenoptic endoscope of claim 17, where the oneor more processing devices is configured to generate representations ofa depth map based on the surface image data and a photographic image.19. A method of presenting an image on a display of a plenopticendoscope, the method comprising: receiving, at an image sensor, lightfrom a target imaging region, wherein the light is captured using aplurality of microlens elements disposed at a sensor end of a fiberbundle comprising a plurality of fiber optic strands, wherein theplurality of microlens elements and the image sensor together form aplenoptic camera to capture both intensity and direction information ofthe light; computing, based on an output of the image sensor, arepresentation of a portion of the target imaging region; computing,based on the output of the image sensor, a depth map associated with theportion of the target imaging region; and presenting the image on thedisplay of the plenoptic endoscope, the image being generated from therepresentation of the portion and the associated depth map associatedwith the target imaging region.
 20. The method of claim 19, wherein theimage is a multi-focus image.